Systems, Devices And Methods For Improved Visualization And Control Of Remote Objects

ABSTRACT

Systems, devices and methods for controlling remote devices by modification of visual data prior to presentation to a person in order to make the person&#39;s response effectively the same as if the person were responding to data transmitted, processed and acted on instantaneously are disclosed. The systems, devices and methods advantageously minimize or eliminate the risks caused by a human response to data that has been delayed in transmission, processing and presentation. In an embodiment, a person controlling a remote device using an augmented reality interface is able to control the device based on predicted positions of an object at the time action is taken, thereby advantageously compensating for delays in receiving data, acting on such data and transmitting instructions or a response to the remote device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 61/973,099 filed Mar. 31, 2014. The text and contents ofthat provisional patent application are hereby incorporated into thisapplication by reference as if fully set forth herein.

FIELD OF INVENTION

The subject disclosure generally relates to the field of augmentedreality (“AR”). Specifically, embodiments of the present inventionrelate to systems, methods and devices for effectively controllingremote objects and/or devices by compensating for the delay in humanperception and response time.

DISCUSSION OF THE BACKGROUND

For the purposes of this specification, the present invention willgenerally be described in relation to controlling drones by predictingthe positions of objects based on delays in human perception andresponse time. However, the invention is not so limited and may beapplied in a variety of other applications where effectively eliminatingthe delays in perception and response time may be beneficial including,but not limited to, online gaming, remote surgery, remote roboticdevices, control or response to elements moving faster than anticipatedby biologically calibrated flash-drag compensation, and other remoteand/or rapidly moving situations and/or targets.

Human sensory processing is not instantaneous. There are significantdelays between generation of data (e.g., photons bouncing off of anobject), reception of the data (e.g., photons reaching the retina),transmission of the data via nerves, and processing of the data by thebrain.

Looking at a specific example, the time between photons bouncing off ofan object in Iraq, being received in a drone camera, being processed bythe drone camera and computer system, and encryption of the data,represents hundreds or thousands of milliseconds of delay. Transmissionto a geostationary satellite and retransmission from the satellite to aNevada ground station takes approximately 250 milliseconds. Decryption,processing and presentation to a drone pilot at that ground stationtakes additional milliseconds. It would not be unusual for the delaybetween generation of data in Iraq and action by a Nevada-located pilotbased on that data to add more than one second to the time between datageneration and implementation of the instructions sent to the drone inresponse.

Looking at a biological example, delays in human data processing meanthat a bird in flight is slightly forward of the position it was in atthe time of generation of the data that underlies the human perceptionof the bird (the time at which the visual data was generated by lightreflecting off of the bird). Evolution provides humans with a brainfunction known as the “flash-drag effect,” whereby the mammalian brainalters the apparent position of perceived objects to match the positionthey will be in at the time the brain is presented with the datareceived in the retina. As a result, for example, a hunter will “see” aflying bird at a point forward of where the photons hitting the retinaactually reveal the bird to be located. Without the flash-drag effect,human hunters would constantly be shooting at a prior position of amoving target.

The flash-drag effect is calibrated for the circumstances for which itevolved—namely, unaided observation of objects in the direct field ofview. As a result, it is unable to compensate for the additional delayscaused by processing of data generated at a distance and/or in anon-biological manner. Similarly, the compensation mechanisms evolved ata time when targets and threats moved at a far slower speed and in adifferent manner than modern targets and threats. For example, aspeeding car travels faster than the 65 miles per hour of the fastestland animal, the cheetah. The benefits of the flash-drag effect weresignificant enough that the effect evolved and persisted. However, it isless efficacious in modern world, and there is a strong need for atechnology that allows similar benefits to be enjoyed with regard tomodern threats, modern targets, non-biologically generated data, andactions over a large distance.

As AR technology becomes more widely adopted as a mechanism for controlof or interaction with remote devices (and in some applications, evenwith respect to control of local devices), compensation for delays intransmission and processing of data, together with mechanisms to addressinstances where the compensation turns out to be inaccurate orinsufficient, will be of enormous value in improving the range andutility of AR interfaces.

Consequently, there is a strong need for systems, methods and devicesthat effectively control remote devices by compensating for the delay inhuman perception and response time. To this end, it should be noted thatthe above-described deficiencies are merely intended to provide anoverview of some of the problems of conventional systems, and are notintended to be exhaustive. Other problems with the current state of theart and corresponding benefits of some of the various non-limitingembodiments may become further apparent upon review of the followingdescription of the invention.

SUMMARY OF THE INVENTION

Embodiments of the present invention relate to systems, methods anddevices for controlling remote devices. In embodiments of the presentinvention, signal processing is used to modify visual data prior topresentation to a person, and/or to modify the action taken based on theresponse of a person, in order to make the human response effectivelythe same as if the human were responding to data transmitted, processed,and acted on instantaneously. This technology negates the risks andimpairments in human response caused by the delays in signaltransmission, processing, and presentation.

The invention brings the benefits of the flash-drag effect to humanperception of types of visual information where the flash-drag effectwould be beneficial, but where the biological implementation offlash-drag is absent, insufficient, or miscalibrated. In one aspect,humans controlling remote devices using an augmented reality interfaceare able to control those devices based on the likely and/or predictedposition of objects at the time action is taken, thereby compensatingfor delays in receiving the data, acting on the data, and receipt, bythe remote object, of the instructions from the operator.

In one embodiment, the invention relates to a method of compensating forthe delay in controlling a remote device, the method comprising (i)analyzing a data stream to identify a moving object, (ii) determiningthe likely trajectory and/or speed of the moving object, (iii)calculating an amount of time between generation and perception of thedata stream by a human, (iv) predicting one or more positions of themoving object based on the amount of time and the trajectory and/orspeed, and (v) displaying the object in the predicted position(s). Insome embodiments, the method may further comprise calculating aprobability that the moving object will be in the predicted position,and if the probability of two or more predicted positions is greaterthan a threshold probability, displaying the predicted positions to thesame person or, alternatively, displaying each of the predictedpositions to a different person.

The invention also relates to a system for compensating for the delay incontrolling a remote device comprising a computing device operablycoupled to the remote device, wherein the computing device is configuredto determine a predicted position of a moving object based on an amountof time between generation of a data stream and perception of the datastream by a human, and the trajectory and/or speed of the moving object.In some embodiments, the predicted position is adjusted to compensatefor encryption and decryption times of the data stream. The predictedposition may also be further adjusted to compensate for processing timeof the human, nerve signal to limb movement conversion time, and thetransmission time of a control input, instruction or response from thehuman to the remote device(s).

The benefits and advantages of the present invention are plenary. Putsimply, the benefits and advantages that led to mammalian evolution ofthe flash-drag effect, are imprecisely analogous to (but a crudeapproximation of a subset of) the benefits and advantages that thepresent invention brings to human interaction with visual data that isnot of a type that triggers an accurate biological flash-drag effect.Further, embodiments of the present invention include correctivetechnology to verify that the predicted path of an object was in factfollowed, and through other aspects described herein, the inventionbrings additional benefits over the biological flash-drag effect.

These and other advantages of the present invention will become readilyapparent from the detailed description below.

BRIEF DESCRIPTION OF THE DRAWINGS

Various non-limiting embodiments are further described with reference tothe accompanying drawings in which:

FIG. 1 schematically illustrates a method for predicting the position ofa moving object, according to an embodiment of the present invention.

FIG. 2 schematically illustrates a system for controlling remote devicesbased on two predictions of the position of a target, according to anembodiment of the present invention.

FIG. 3 schematically illustrates a method for control of remote devicesbased on a predetermined tolerance for the predicted position of anobject, according to an embodiment of the present invention.

FIG. 4 schematically illustrates a method for control of remote objectsby transmitting provisional instructions based on the predicted humanresponse to the position of a moving object, according to an embodimentof the present invention.

FIG. 5 schematically illustrates a method for control of remote objectsbased on a display of multiple predicted positions of an object.

DETAILED DESCRIPTION

Reference will now be made in detail to various embodiments of theinvention, examples of which are illustrated in the accompanyingdrawings. While the invention will be described in conjunction with thefollowing embodiments, it will be understood that the descriptions arenot intended to limit the invention to these embodiments. On thecontrary, the invention is intended to cover alternatives,modifications, and equivalents that may be included within the spiritand scope of the invention as defined by the appended claims.Furthermore, in the following detailed description, numerous specificdetails are set forth in order to provide a thorough understanding ofthe present invention. However, it will be readily apparent to oneskilled in the art that the present invention may be practiced withoutthese specific details. In other instances, well-known methods,procedures and components have not been described in detail so as not tounnecessarily obscure aspects of the present invention. Theseconventions are intended to make this document more easily understood bythose practicing or improving on the inventions, and it should beappreciated that the level of detail provided should not be interpretedas an indication as to whether such instances, methods, procedures orcomponents are known in the art, novel, or obvious.

There are a variety of circumstances in which humans are expected toprocess and/or act on data that was generated a material number ofmilliseconds prior to the time at which the human receives the data andthe brain processes it in a manner sufficient to permit action.

Without limiting the invention to these applications, it is useful tonote that humans may experience a material delay between generation ofvisual data and processing of that data in a variety of circumstancesthat may include, among others, online gaming, control of drones, remotesurgery, remote robotic devices, control and/or response to elementsmoving faster than anticipated by biologically calibrated flash-dragcompensation, and other remote and/or rapidly moving situations and/ortargets. In one aspect, the instant invention may utilize one or moreapproaches to compensate for this delay.

One aspect is to utilize digital signal analysis and processing toidentify objects in a video stream, determine their most likelytrajectory and speed (whether an instant trajectory and speed, atrajectory and speed as it is most likely to change during the relevantperiod, or a combination thereof), calculate the amount of time that ithas taken between original generation of the data and the calculation,calculate the amount of time it is most likely to take between thecalculation and the conversion of the data to a human brain's perceptionof the object represented by the data, and then modify the data so thatthe object is shown, in the data viewed by the human, to be in aposition that it will most likely actually be in at the time of humanperception (typically forward of its actual position at the time ofobservation).

FIG. 1 schematically illustrates a method 100 that compensates for thedelay in controlling a remote device utilizing digital signal analysisto display an object in a position the object will likely be in at thetime of perception by a person. The method begins at step 110 wherein adigital data stream is analyzed to identify a moving object within thestream. The data stream may be analyzed by using digital signalanalysis, object recognition algorithms and/or other data analysis andprocessing methods to identify such moving objects. At step 120, andbased on such analysis, the speed and/or likely trajectory of the objectis determined. At step 130, the time between the generation of the datastream and the perception of the object by a person is calculated.

Although in the embodiment of FIG. 1, the calculation at step 130 isshown to be a single step, the calculation may be performed in multiplesteps. For example, an amount of time between the original generation ofthe data stream and when the calculation is performed may first becalculated, and then subsequently, a second calculation may be performedto determine an amount of time between the calculation and theconversion of the data to the person's perception of the object in thestream. Additionally, such calculations may be performed in any order.

At step 140, one or more predicted positions of the object in the datastream are determined based on the likely speed and trajectory of theobject. In some embodiments, the prediction position(s) may be madebased on the instantaneous speed and/or trajectory of the object, whilein other embodiments, the predicted position(s) may be made based on therate of change of the object's speed and/or trajectory. In yet otherembodiments, a combination of instantaneous speed and/or trajectory andthe rate of change of the speed and/or trajectory may be used to predictthe position(s) of the object. At step 150, the data stream is alteredand/or modified to display the object in the predicted positions(s). Itshould be noted that more than one position of the object may be likelybecause of the physical characteristics of the object and/or path onwhich the object is travelling (e.g., the object may be travelling on aroad nearing an intersection, the object may be slowly, perhaps toreverse course, etc.).

In one aspect, the predicted position of an object may includecompensation for the human biological flash-drag effect. In implementingsuch compensation, the object may be displayed to a human in a positionwhere the flash-drag effect will then further change the perceivedlocation of the object, and together with the original, digitalalteration of object position, enable the human to perceive the objectin its most likely actual position at the time the human perceptiontakes place.

Where an analysis of the data stream indicates multiple likely positionsat the time of human perception (for instance, where more than oneposition exceeds a threshold probability), the data may be digitallyprocessed for viewing by more than one human operator, with eachoperator receiving data reflecting a different predicted position.Alternatively, the multiple predicted positions may be displayed to oneperson.

Many aspects of the invention are best disclosed and understood bydiscussion in the context of an exemplary use. In this document, controlof a drone by a human operator is one of the primary settings for theexample. However, it should be understood that applications of theinvention may extend to many other uses.

Consider the simple example of a target running down a path asillustrated in FIG. 2. In the system 200 of FIG. 2, path 202 has a leftfork 203 and a right fork 204. When the speed of the target isdetermined by analysis of the data stream, and the amount of timebetween the generation of the data stream and human perception iscalculated, the target 201 may travel beyond the intersection of theleft fork 203 and the right fork 204. Consequently, there may be twopossible positions of the target 201 at the time of perception: firstpredicted position 251 and second predicted position 252. One humanoperator 206 may be shown the target 201 travelling down the left fork203 to predicted position 251, and a second human operator 207 may beshown the target 201 travelling down the right fork 204 to predictedposition 252.

Typically, after a human is presented with the altered data (thepredicted position or positions are displayed), the human may send acontrol signal, instruction or a response to the altered data.Continuing with the example of the target 201 approaching a fork in theroad, the first human operator 206 who received the first predictedposition 251 (the “left” data) may instruct a first drone 261 to launcha first missile 263 (e.g., a full payload missile) to the firstpredicted position 251 (e.g., a position 100 yards forward on the leftfork 203) by transmitting a first control signal 208, while the secondhuman operator 207 who received the second predicted position 252 (the“right” data) may instruct a second drone 262 to launch a second missile264 (e.g., a reduced payload/reduced “kill zone” missile because, forexample, there are civilian structures 205 closer to the right fork 204than to the left fork 203) to the second predicted position 252 (e.g., aposition 100 yards forward on the right fork 204) by transmitting asecond control signal 209.

In the embodiment of FIG. 2, a first computing device 265 may comparethe first predicted position 251 to an actual position of the target(not shown), and a second computing device 266 may compare the secondpredicted position 252 to the actual position of the target. In someembodiments, the computing devices 265, 266 may be computers operablycoupled to, respectively, the first and second drones 261, 262 (thedevices receiving the instructions from the human operators). However,in other embodiments the computing devices 265, 266 may be anotherdevice or other person(s) receiving the instructions and comparing theactual position(s) of the target to the predicted position(s) that weretransmitted to the humans 206, 207 sending the first and second controlsignals 207, 208.

In one aspect, where the data sent to a human operator (e.g., the humanoperator 206 or 207 in FIG. 2) includes a predicted position thatmatches the actual observed target position (in some instances, towithin a predetermined and/or pre-specified tolerance), the instructionsor control signal sent by that human (e.g., the control signal 208 or209 in FIG. 2) are followed. The requisite tolerance may vary dependingon the importance of taking some action, the costs of taking no action,or based on other criteria. In some aspects, if the predicted positiondoes not match the actual observed target position within thepredetermined/pre-specified tolerance, then the instruction, controlsignal or response sent by the human may be modified based on the actualposition.

FIG. 3 schematically illustrates a method 300 wherein a human responseis modified based on a comparison of the actual position to thepredicted position of the object. The method begins in a similar mannerto the method of FIG. 1. At step 310, a data stream is analyzed toidentify a moving object. At step 320, the likely trajectory and/orspeed of the object are determined, and at step 330, the time betweengeneration of the data stream and human perception is calculated. Atstep 340, the predicted position of the moving object is determinedbased on the likely speed and/or trajectory of the object and thecalculated time.

From step 350 on, the method 300 of FIG. 3 differs from the method 100of FIG. 1, because at step 350, the predicted position of the object iscompared to the actual position of the object. In some instances, theactual position of the object may be determined by computers operablycoupled to the device receiving the control signal, instructions orresponse from the human operator, while in other instances, thecomputing device may be another device or other person receiving theinstructions and comparing the actual position(s) of the target to thepredicted position. At step 355, a determination is made as to whetherthe predicted position of the object is within a pre-determinedtolerance of the actual position. The predetermined tolerance may bepreset based on a percentage of the distance traveled by the object, afinite error rate, the speed at which the object is traveling, etc. Thetolerance may also vary based on the type of object, the cost of theresponse, the potential damage caused by an inaccurate response, etc.

If the predicted position of the object when compared to the actualposition is within the predetermined/pre-specified tolerance, at step360, the control signal, instructions or response from the humanoperator for control of the remote device may be followed. If thepredicted position of the object when compared to the actual position isnot within the predetermined/pre-specified tolerance, at step 370, thehuman operator's control signal, instruction or response may be modifiedbased on the actual position of the object. In some instances, the humanoperator's control signal, instruction or response may be aborted orrejected.

In one aspect, the predicted position of the target may also be adjustedto account for the time it will take for the instructions to get fromthe human to the device that the human is controlling or instructing.Returning to the example of the left/right forks (e.g., left and rightforks 203, 204 of FIG. 2), if it takes 1,000 milliseconds for the signalto travel from the human operator (e.g., human 206, to the computer thatencrypts it (e.g., computing device 265 of FIG. 2), from that computerto a satellite (not shown in the figures), from the satellite back tothe drone (e.g., first drone 261 of FIG. 2), and to be decrypted by thedrone, the predicted position of the target (e.g., target 201 in FIG. 2)may be displayed to the human operator based on an additional 1,000milliseconds of trajectory and velocity.

To further illustrate the problem in the context of control of a drone,there is a delay of approximately 250 milliseconds second to go fromIraq to Nevada via geostationary satellite, and 250 milliseconds for thereturn trip. Added to this is time for encryption/decryption, processingtime for the human, and time for a nerve signal to be converted tomechanical energy of a limb. Further, humans perceive items where theywill be in around 250 milliseconds as a basic brain function (i.e., theflash-drag effect) and as a result, approximately 250 milliseconds maybe subtracted from the predicted delay. The data coming from the dronewould then be digitally manipulated to change target location and/orspeed to compensate for these or other additional delays.

In one aspect, delays may be reduced by intercepting nerve signals(e.g., by using electrodes) at a point above the limb, or at least abovethe hand or foot, and converting the nerve signals into computer inputsin a manner crudely analogous to the difference between mechanical andfly-by-wire aircraft control.

It should be appreciated that it takes a substantial amount of time toconvert a thought into mechanical movement of a finger or otherappendage that moves or otherwise manipulates a control surface. Acomputer utilizing a Bayesian or other artificial intelligence orlearning algorithm may determine signals that a particular operatorsends indicating a likely muscle movement.

For example, an operator may experience a particular brainwave pattern80% of the time prior to hitting “launch” on a drone missile. Similarly,an operator may experience an increase in electrical conduction by theskin, indicating an increase in sweating, 50% of the time when theoperator is about to hit “abort” to call off or disable a missile.Indeed, any of the numerous “tells” that are observed by poker players,FBI agents and others who are trained to identify the significance ofsmall signals may be correlated with certain actions by a particularoperator.

In order to reduce delay that occurs in conversion of a thought to afully actuated, muscle-driven control input, control signals may be sentto a remote device based on exceeding a confidence threshold inpredicting an operator's response based on “tells”. In one aspect, theinstruction may simply be sent. In another, the instruction may be sentin a provisional manner, initiating the transmission, encryption,reception, decryption, and movement of the drone (or other remotedevice) in preparation for the predicted action. When the actual controlinput is made (or, in one aspect, when sufficient control input is madeto exceed a threshold likelihood that the user is in fact making thepredicted control input), a signal is sent confirming the priorinstruction.

In another aspect, if the confidence threshold does not exceed thethreshold level, then the instruction is sent provisionally and isconfirmed when the actual response is received. In one aspect, theconfirming signal may be sent in an expedited manner, for example, bybypassing the encryption. Whether sent in an expedited manner or not,upon receipt of the confirming signal, the remote device would alreadybe positioned to act on the instructions more rapidly than if theprovisional signal had not been sent.

In another aspect, if no confirming signal is sent and no rejectingsignal is sent, the action may be completed, aborted, or acted uponbased on whether the confidence level in the “tell” exceeds a setthreshold. In another aspect, if the operator does not make the expectedcontrol input, an “abort” or “rejecting” signal may be sent in ananalogous manner to the confirming signal, and the action aborted.

Referring now to FIG. 4, an exemplary method 400 for reducing the delaythat occurs in conversion of a thought to a control input isschematically shown. The steps 410 through 440 of method 400 areeffectively the same steps 110 through 140 of method 100 of FIG. 1, andmethod 300 of FIG. 3. That is, at step 410, the data stream is analyzedto identify a moving object. At step 420, the likely trajectory and/orspeed are determined. At step 430, the time between generation of thedata stream and perception by the human operator are calculated, and atstep 440, the predicted position of the moving object is determined.

At step 450 of method 400, the control input, instructions or responseof the human operator is predicted based on the human operator's “tells”as described above. At step 460, the confidence level of the predictedcontrol input, instructions or response is calculated. At step 465, theconfidence level of the predicted control input, instructions orresponse from the human operator is compared to a predeterminedconfidence level. The requisite confidence level may vary depending onthe importance of taking some action, the costs of taking no action, thetype of moving object, or based on other criteria.

In the exemplary method 400 of FIG. 4, if the confidence level isgreater than the threshold level, then at step 470, the predictedcontrol input, instructions or response is implemented. If theconfidence level does not exceed the threshold level, then at step 480,a provisional control input, instruction or response is transmitted. Atstep 490, the actual response is received, and at step 495, adetermination is made as to whether the actual response is the same asthe predicted response. If yes, then at step 496, the control input,instruction or response is transmitted in an expedited manner. If no,then at step 497, the provisional control input, instruction or responseis aborted, and the actual response is implemented.

Further elaborating on the drone example, consider a drone operator inNevada controlling a drone flying over Iraq who is presented with datathat is one second old. The operator may take one second to perceive,process, and react to the data, and the transmission of the response tothe drone may take one second in transmission. In one aspect, theembodiments herein alter the data presented to the human operator in away that shows the scene as it will exist at the time that the humanresponse reaches the drone, allowing the most accurate and appropriateresponse by the human. In some aspects, when the human response isdelayed more than the system anticipated, the response inputs may bealtered to compensate for the movement of the objects being acted uponduring that extra period of delay.

Consider a situation where the data presented to the operator isdetermined to be wrong, for example, if the target was predicted to be,and therefore is shown as being, in a position 50 meters north of itsactual position three seconds after the time of measurement, butunexpectedly only travelled 30 meters in those three seconds. In thatexample, the control inputs sent by the remote operator may be negated,altered to take account of the different position, or otherwise modifiedor evaluated. In certain aspects, the effective range of the device (forexample, the kill range of a piece of ordnance) may be utilized todetermine whether and to what extent the control instructions should bealtered or negated. In either event, the instant inventions make remotecontrol and instruction more accurate than existing methods.

It should be noted that human control input may be utilized inconjunction with artificial intelligence or other programming of theremote device, so that even if the data presented to the human is lessaccurate than a desired threshold, the device may be able to compensatefor the inaccuracy in the data presented to the human by modifying thehuman's instructions in a manner consistent with the error in the datapresented to the human.

Returning to the left/right fork for example (see e.g., FIG. 2), imaginethat there is only a single human operator, and the system detects thatthe target (e.g., target 201 of FIG. 2) is leaning slightly left whenapproaching the intersection of the left and right forks (e.g., forks203 and 204 of FIG. 2). The predicted path is thus that the target goesleft (takes left fork 203 of FIG. 2), and that is the visual imagepresented to the human operator. The human operator observes the sceneand sends a control input to launch a missile (e.g., first missile 263of FIG. 2) at the target. When the control input is received at theremote drone (e.g., first drone 261 of FIG. 2), the drone observes thatthe target actually took the right fork (e.g., right fork 204 of FIG.2). The drone may, in one aspect, analyze the environment and otherelements in the predicted left path and the actual right path anddetermine that the elements are more than X % similar, that there is anabsence of Y type elements in both forks, and/or that it is more than N% confident that the human operator would have made the same decisionhad the prediction accurately shown the human operator that the targettook the right path. If the confidence level exceeds a threshold, thedrone may recalculate the human input for the left fork to deliver thesame response to the target in the actual, right fork position.

In another aspect, where there are more possible predicted positionsthan the number of human operators, each operator may be shown a rangeof possible positions. Returning to the example of the left/right fork,imagine a single human operator. The operator may be shown the targetsimultaneously in both positions and, via appropriate input devices suchas a touch-screen; the operator may indicate whether the left, right, orboth positions should be fired upon.

Referring now to FIG. 5, some aspects of the present invention areschematically illustrated. FIG. 5 depicts a scene as it might be viewedby a drone flying overhead. The drone would transmit the scene to ahuman operator, await instructions from the operator, and implement theinstructions when received. In FIG. 5, a car 500 containing a target(for example, a terrorist) may be observed moving at a certain rate ofspeed. As the video (which may, in one aspect, be a single frame, but inother aspects would be a stream of frames) is processed, the speed ofthe vehicle together with the paths determined to be most likely pathsfor the car to take, together with the most likely locations of the car,may be calculated. In FIG. 5, the car 500 may go into reverse, followinga reverse path 501, may continue forward on a forward path 503, may turnleft 502 toward a house 506, or may turn right 504, somewhat proximateto a house 505. While the forward path 503 is the most likely, the otherpaths 501, 502, 504 are also possible.

The operator may be presented with the single most likely location ofthe vehicle at the time his input is acted upon; however, it may bedesirable to present the operator with a plurality of possiblelocations, in one aspect, those locations that exceed a thresholdlikelihood. Using the probabilities described above with regard to FIG.5, the relative likelihood of various vehicle locations may be depictedin a manner that the human operator can perceive. FIG. 5 illustrates onemethod, namely, indicating likelihood of various positions by providinga darker (optionally colored) pathway. Another method illustrated byFIG. 5 is in providing arrows with increasingly large sizes indicatingmore probable pathways. Another method illustrated by FIG. 5 is that theindicated areas, such as the overlaid arrows 501, 502, 503, 504 mayutilize a luminosity, a color and/or other gradient to indicateincreasing or decreasing likelihood that the vehicle will be in thatposition at the target time.

In one aspect, the human operator may indicate which of the pathwaysshould undergo which response, for example by touching, on a touchscreen, pathways that should be fired upon with a missile. In FIG. 5,the operator may touch the two arrows 503, 504 that do not intersectwith a house, indicating that when the instructions are received at thedrone, if the vehicle has taken (or, in one aspect, if the vehicle latertakes) one of the indicated pathways 503, 504, the vehicle should befired upon, while the vehicle should not be fired upon if it does nottake the indicated pathways.

The invention, while well illustrated in aspects by the example of adrone, is not so limited. Take for example a remote surgery application.Consider a surgical robot in Sudan being controlled by a pair of doctorsin Boston. The surgical robot is cutting and a tiny bit of blood isdetected. The computer system determines it is 50% likely to be a cutartery, 50% likely to be capillary blood or a similar artifact. Onesurgeon is shown a cut artery while another is shown nothing unusual.When the surgeons' inputs are received at the device, the data sent tothe surgeons is compared to the then-current data and the surgeon whoreceived the data that is most similar to the then-current data hastheir inputs utilized.

Similarly, if there is a single surgeon, the surgeon may be shown thecut artery, optionally together with a message indicating that it is apossibility of a cut artery or even the approximate risk that the cutartery has happened. The surgeon may make control inputs based on thecut artery scenario. If the artery is indeed cut, at the time thesurgeon's control inputs arrive in Sudan, the device in Sudan determinesthe cut artery data is correct and that the control inputs were based onsufficiently accurate data and utilizes them. If the device determinesthat there was no cut artery, the control inputs are not utilized. Inone aspect, a second surgeon or an artificial intelligence orprogrammatic control system may take action to maintain the status-quoduring the time when the control inputs from Boston were based on aninaccurate prediction.

In another aspect, the image displayed to one or more surgeons (in thesurgical example, or operators in other applications) shows a pluralityof possible outcomes simultaneously. In one aspect, the opacity withwhich the outcomes are displayed is determined, at least in part, by thelikelihood that the proposed action will result in the displayedoutcomes. In another aspect, where the predicted position of an objectincludes more than one possible position, one or more of the possiblepositions may be simultaneously displayed. Optionally, the opacity ofthe object displayed in each of the possible positions may be modifiedto reflect the likelihood that the object will be in that position atthe relevant time. In another aspect, where the predicted trajectoryand/or pathway of object includes more than one likely trajectory orpathway, the opacity of the trajectory and/or pathway displayed may bemodified to reflect the likelihood that the trajectory and/or pathway ofthe object is the actual trajectory or pathway.

Note that while this document refers to visual data, it should beappreciated that audio or other data may also be processed in the mannerdescribed herein. Further, while the document refers to human dataprocessing, it should be appreciated that a non-human animal or even acomputer may, in whole or in part, enjoy certain benefits of the instantinvention.

What is claimed is:
 1. A method of compensating for the delay in controlof a remote device, the method comprising: analyzing a data stream toidentify a moving object; determining the likely trajectory and/or speedof the moving object; calculating an amount of time between generationof the data stream and perception of the data stream by a human;predicting one or more positions of the moving object based on theamount of time and the trajectory and/or speed; displaying the object inthe predicted position(s).
 2. The method of claim 1, further comprisingcalculating a probability that the moving object will be in thepredicted position(s).
 3. The method of claim 2, further comprising, ifthe probability of two or more predicted positions is greater than athreshold probability, displaying a plurality of the predicted positionsto a person.
 4. The method of claim 2, further comprising, if theprobability of two or more predicted positions is greater than athreshold probability, displaying different predicted positions todifferent persons.
 5. The method of claim 4, wherein the opacity of thedifferent predicted positions displayed relates to the likelihood thatthe object will be in the predicted position.
 6. The method of claim 1,further comprising, altering the amount of time to compensate for aflash-drag effect of the human.
 7. The method of claim 1, wherein thelikely trajectory and/or speed are determined based on an instantaneoustrajectory and/or speed, a changing trajectory and/or speed, and/or acombination thereof.
 8. The method of claim 1, further comprisingsending one or more control signals, instructions and/or responses fromthe human to the remote device when an actual position and one of thepredicted positions are the same within a predetermined tolerance. 9.The method of claim 8, further comprising adjusting the predictedposition(s) to compensate for a transmission time for the controlsignal, instruction and/or response from the human to be received by theremote device.
 10. The method of claim 1, further comprising negating oraltering a control signal, instruction and/or response from the humanwhen an actual position and one of the predicted positions is less thana predetermined tolerance.
 11. A system for compensating for the delayin controlling a remote device, the system comprising: a computingdevice operably coupled to the remote device; wherein the computingdevice is configured to determine a predicted position of a movingobject based on an amount of time between generation of a data streamand perception of the data stream by a human, and the trajectory and/orspeed of the moving object.
 12. The system of claim 11, wherein thepredicted position is adjusted to compensate for encryption anddecryption times of the data stream.
 13. The system of claim 11, whereinthe predicted position is adjusted to compensate for processing time ofthe human and nerve signal to limb movement conversion time.
 14. Thesystem of claim 13, wherein the predicted position is further adjustedto compensate for a transmission time of a control input, instructionand/or response from the human to the remote device(s).
 15. The systemof claim 11, wherein the computing device is further configured todetermine a predicted response of the human.
 16. The system of claim 15,wherein the computing device is further configured to calculate aconfidence level of the predicted response of the human, and when theconfidence level is greater than a threshold level, instruct the remotedevice(s) to implement the predicted response.
 17. A system forcompensating for the delay in controlling a remote device, the systemcomprising: a computing device operably coupled to the remote device;wherein the computing device is configured (i) to determine a predictedposition of a moving object based on an amount of time betweengeneration of a data stream and perception of the data stream by ahuman, and the trajectory and/or speed of the moving object, and (ii) totransmit provisional instruction(s) to the remote device(s) to implementthe predicted response.
 18. The system of claim 17, further comprisingconfirming the provisional instructions in an expedited manner when acalculated confidence level of the predicted movement is greater than athreshold level.
 19. The system of claim 18, wherein the expeditedmanner comprises bypassing encryption and decryption of the data stream.20. The system of claim 17, wherein if the actual response of the humanis not the predicted response, the provisional instructions to implementthe predicted response is aborted.